In [26]:
pd.Series(y_train_t).hist(figsize=(15, 5))
pd.Series(y_test_t).hist(figsize=(15, 5))
plt.show()
Insurance companies provide coverage for expenses incurred by policyholders due to damages to health or property. The policies they commonly offer encompass medical bills, home, motor vehicle, and fire insurance, as well as financial losses like income loss, all in exchange for a fee or premium paid by the client. Conventional methods of calculating premiums necessitate significant manual labor, and their complexity is continuously growing to account for the intricate data interactions.
Insurance companies typically aim to collect a premium higher than the sum they pay to the insured individual in the event of a valid claim, ensuring profitability. Profitability is the linchpin for the insurance firm's sustainability, thus prompting the need for a dependable mechanism to forecast healthcare expenditures.
Consequently, our objective revolves around constructing a machine learning model to aid in setting rates by predicting the charges or payouts facilitated by the health insurance firm, thereby maintaining profitability.
In this project, our primary focus lies in constructing an XGBoost Regressor to ascertain healthcare expenses based on features such as age, BMI, smoking habits, and more. We'll delve into understanding categorical correlations, building a linear regression model as a baseline, and subsequently comparing it to the outcomes of the XGBoost regressor. Ultimately, we'll acquire the skill of effectively conveying technical outcomes to non-technical stakeholders
Pandas: pandas is a fast, powerful, flexible, and easy-to-use open-source data analysis and manipulation tool built on top of the Python programming language. Refer to documentation for more information.
NumPy: The fundamental package for scientific computing with Python. Fast and versatile, the NumPy vectorization, indexing, and broadcasting concepts are the de-facto standards of array computing today. NumPy offers comprehensive mathematical functions, random number generators, linear algebra routines, Fourier transforms, and more. Refer to documentation for more information. pandas and NumPy are together used for most of the data analysis and manipulation in Python.
Matplotlib: Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. Refer to documentation for more information.
scikit-learn: Simple and efficient tools for predictive data analysis accessible to everybody and reusable in various contexts. It is built on NumPy, SciPy, and matplotlib to support machine learning in Python. Refer to documentation for more information.
statsmodels: statsmodels is a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests and statistical data exploration. Refer to documentation for more information.
plotly: Plotly's Python graphing library makes interactive, publication-quality graphs. Examples of how to make line plots, scatter plots, area charts, bar charts, error bars, box plots, histograms, heatmaps, subplots, multiple-axes, polar charts, and bubble charts. Refer to documentation for more information.
scikit-optimize: Scikit optimize enables sequential model based optimization in Python. Refer to documentation for more information.
category_encoders: A set of scikit-learn-style transformers for encoding categorical variables into numeric with different techniques. Refer to documentation for more information.
xgboost: XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable. It implements machine learning algorithms under the Gradient Boosting framework. XGBoost provides a parallel tree boosting (also known as GBDT, GBM) that solve many data science problems in a fast and accurate way.
import pandas as pd
import numpy as np
import plotly.express as px
import matplotlib.pyplot as plt
import sys
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression
import statsmodels.api as sm
import math
from xgboost import XGBRegressor
from sklearn.pipeline import Pipeline
from skopt import BayesSearchCV
from skopt.space import Real, Categorical, Integer
from sklearn.preprocessing import StandardScaler, PowerTransformer
from sklearn.feature_selection import RFE
And the functions in the ml_pipeline folder:
sys.path.append("ml_pipeline/")
from eda import plot_histograms, plot_univariate_numeric, plot_univariate_categorical, plot_heatmap, plot_paired_boxplots, plot_paired_scatterplots, plot_residuals, plot_pearson_wrt_target
from stats import chi2, anova
from model_performance import calc_model_performance, compare_model_performance, calc_preds_in_residual_range, calc_preds_in_residual_perc_range
Exploratory Data Analysis, commonly referred to as EDA, is a methodology employed to dissect data through visual representation. This approach involves utilizing both statistical measures and visual techniques to pinpoint specific trends within the dataset.
EDA serves the purpose of comprehending data patterns, identifying anomalies, validating assumptions, and more. Its primary objective is to provide insights into the data landscape before forming any hypotheses.
In the realm of constructing a machine learning model, EDA stands as a crucial initial step. It grants us the capacity to grasp the distribution and interrelations of variables, as well as to gauge the potential predictive strength of various features.
Let us commence by importing the dataset, which is stored within the insurance.csv file located in the input folder:
data = pd.read_csv("data/insurance.csv")
data.head()
| age | sex | bmi | children | smoker | region | charges | |
|---|---|---|---|---|---|---|---|
| 0 | 19 | female | 27.900 | 0 | yes | southwest | 16884.92400 |
| 1 | 18 | male | 33.770 | 1 | no | southeast | 1725.55230 |
| 2 | 28 | male | 33.000 | 3 | no | southeast | 4449.46200 |
| 3 | 33 | male | 22.705 | 0 | no | northwest | 21984.47061 |
| 4 | 32 | male | 28.880 | 0 | no | northwest | 3866.85520 |
data.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1338 entries, 0 to 1337 Data columns (total 7 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 age 1338 non-null int64 1 sex 1338 non-null object 2 bmi 1338 non-null float64 3 children 1338 non-null int64 4 smoker 1338 non-null object 5 region 1338 non-null object 6 charges 1338 non-null float64 dtypes: float64(2), int64(2), object(3) memory usage: 73.3+ KB
NOTE: Absence of null values across all columns negates the necessity for value imputation during the Data preprocessing phase. Nevertheless, this is typically a facet that warrants attention while constructing a machine learning model.
The target (i.e., the variable we aim to forecast) is the charges column; hence, let's bifurcate the dataset into features (X) and the target (y):
target = 'charges'
X = data.drop(target, axis=1)
y = data[target]
X.shape, y.shape
((1338, 6), (1338,))
plot_histograms(X)
Key observations regarding the distribution of individual features:
age - Displays an approximately uniform distribution.sex - Demonstrates nearly equal volume across each category.bmi - Exhibits an approximately normal distribution.children - Shows right skewness, indicating a greater volume in the lower range.smoker - Evidences significantly higher volume in the no category compared to the yes category.region - Presents an approximately balanced volume in each category.plot_histograms(pd.DataFrame(y), height=300)
The distribution is right skewed (i.e. higher volume in the lower range).
plot_univariate_numeric(X.select_dtypes(include="number"), y)
Points to note regarding each feature:
age - As age increases, charges also tends to increase (although there is a large variance in charges for a given age).bmi - There is no clear relationship, although there seems to be a group of individuals with bmi > 30 that have charges > 30k. This group may become more apparent when we carry out our bivariate analysis later.children - No clear relationship (although charges seems to decrease as children increases). Since there are only 6 unique values for this feature, let's try treating it as a categorical feature for the purposes of univariate analysis.plot_univariate_categorical(X[['sex', 'smoker', 'region', 'children']], y)
Points to note regarding each feature:
sex - No significant differences in charges between the categories.smoker - charges for smoker == 'yes' are generally much higher than when smoker == 'no'.region - No significant differences in charges between the categories.children - No significant differences in charges between the categories (children >= 4 are skewed towards lower charges, but this is likely due to the low volumes in those categories - see the Distributions section).plot_heatmap(X[['age', 'bmi', 'children']], y, bins=10)
No additional insight can be gained from these plots that we haven't already obtained from the univariate analysis.
plot_paired_boxplots(X[['sex', 'smoker', 'region']], y)
Points to note regarding these feature pairs:
sex-smoker - The median charges is higher for males who smoke compared to females who smoke (36k vs 29k)smoker-region - The median charges is higher for smokers in the southwest and southeast vs the northeast and northwest (35k and 37k vs 28k and 27k)plot_paired_scatterplots(X, y)
There are two pieces of insight from these graphs:
age-smoker - There is a group in the bottom left hand quadrant of the chart (where age < 50 and smoker = 'no') where all beneficiaries have health care costs below 10k (which is relatively small compared to the rest of the population).bmi-smoker - There is a group in the top right hand quadrant of the chart (where bmi > 30 and smoker = 'yes') where all beneficiaries have health care costs above 30k (which is relatively large compared to the rest of the population).px.scatter_matrix(
X.select_dtypes(include=np.number)
)
There doesn't look like there's much correlation between any of the numeric features. To be sure, let's calculate and plot the Pearson's correlation matrix:
px.imshow(X.select_dtypes(include=np.number).corr())
This suggests there is very little correlation between the numeric features - the highest being a Pearson's correlation of 0.11.
X_chi2 = chi2(X.select_dtypes(object))
X_chi2
| column1 | column2 | chi_squared | p_value | dof | |
|---|---|---|---|---|---|
| 0 | sex | smoker | 7.392911 | 0.006548 | 1 |
| 1 | sex | region | 0.435137 | 0.932892 | 3 |
| 2 | smoker | region | 7.343478 | 0.061720 | 3 |
A threshold of < 0.05 is widely accepted for rejecting the null-hypothesis (that the features are independent) since this means that there is only a 5% probability of observing the distribution by chance.
The only feature pair with a p-value less than this threshold is sex and smoker, which means it's likely that these features are correlated:
X_anova = anova(X)
X_anova
| num_column | cat_column | f_stat | p_value | |
|---|---|---|---|---|
| 0 | age | sex | 0.581369 | 4.459107e-01 |
| 1 | age | smoker | 0.836777 | 3.604853e-01 |
| 2 | age | region | 0.079782 | 9.709891e-01 |
| 3 | bmi | sex | 2.878970 | 8.997637e-02 |
| 4 | bmi | smoker | 0.018792 | 8.909850e-01 |
| 5 | bmi | region | 39.495057 | 1.881839e-24 |
| 6 | children | sex | 0.393659 | 5.304898e-01 |
| 7 | children | smoker | 0.078664 | 7.791596e-01 |
| 8 | children | region | 0.717493 | 5.415543e-01 |
The only feature pair with a p-value less than this threshold is bmi and region, which means it's likely that these features are correlated:
plot_pearson_wrt_target(X, y)
The children feature has a very low correlation with respect to the target.
data_anova = anova(data)
anova_wrt_target = data_anova[data_anova['num_column']=='charges']
anova_wrt_target
| num_column | cat_column | f_stat | p_value | |
|---|---|---|---|---|
| 9 | charges | sex | 4.399702 | 3.613272e-02 |
| 10 | charges | smoker | 2177.614868 | 8.271436e-283 |
| 11 | charges | region | 2.969627 | 3.089336e-02 |
All p-values are < 0.05, which means that the differences observed in the charges column when comparing the categories within a categorical variable are statistically significant. Note however that this doesn't measure the magnitude of the differences observed.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
cols_to_drop = ['children', 'region', 'sex']
X_train.drop(cols_to_drop, axis=1, inplace=True)
X_test.drop(cols_to_drop, axis=1, inplace=True)
After our EDA, we concluded that the children feature is not strongly correlated to charges. This violates Assumption 2 of a linear regression model, so we should remove this feature from our training set.
Also, our $\chi^2$ test suggested the sex feature is correlated to the smoker feature, and our ANOVA test suggested the region feature is correlated to the bmi feature. This violates Assumption 3 of a linear regression model, so we should remove these features from our training set. We will remove the sex and region features, since these had weaker predictive power with respect to the target.
pt = PowerTransformer(method='yeo-johnson')
y_train_t = pt.fit_transform(y_train.values.reshape(-1, 1))[:, 0]
y_test_t = pt.transform(y_test.values.reshape(-1, 1))[:, 0]
pd.Series(y_train_t).hist(figsize=(15, 5))
pd.Series(y_test_t).hist(figsize=(15, 5))
plt.show()
This code snippet serves the purpose of transforming the target variable using the Yeo-Johnson method through the PowerTransformer class. By applying this transformation to both the training y_train and test data, the code aims to alleviate the impact of non-normality in the data distribution.
The subsequent creation and visualization of histograms provide an overview of the resulting transformed data distribution, offering insights into the effectiveness of the transformation in achieving a more normal distribution. This process can be particularly beneficial for improving the performance of certain machine learning algorithms that assume or benefit from normally distributed data.
from sklearn.compose import ColumnTransformer
cat_cols = ['smoker']
num_cols = [col for col in X_train.columns if col not in cat_cols]
preprocessor = ColumnTransformer(
transformers=[
('cat', OneHotEncoder(), cat_cols)
],
remainder='passthrough'
)
pipeline = Pipeline([
('preprocessor', preprocessor),
('regressor', LinearRegression())
])
pipeline.fit(X_train, y_train_t, regressor__sample_weight=y_train / y_train.min())
Pipeline(steps=[('preprocessor',
ColumnTransformer(remainder='passthrough',
transformers=[('cat', OneHotEncoder(),
['smoker'])])),
('regressor', LinearRegression())])
This code creates a pipeline that combines data preprocessing and modeling for linear regression. The pipeline performs necessary processing on categorical columns (via One-Hot encoding) while leaving numerical columns unchanged. Next, a linear regression is applied to predict the target variable. Sample weights are used to give more importance to certain observations during model training. The overall goal is to build an enhanced linear regression model by handling different data types and accounting for sample weights.
Now that we've trained our model, we can use it to generate predictions on both our training and test sets:
y_pred_train = pipeline.predict(X_train)
y_pred_test = pipeline.predict(X_test)
Remember that the model was trained based on a transformed version of the target. Therefore we need to perform the inverse Yeo-Johnson transform to convert the model predictions to the original format of the target:
y_pred_train = pt.inverse_transform(y_pred_train.reshape(-1, 1))[:, 0]
y_pred_test = pt.inverse_transform(y_pred_test.reshape(-1, 1))[:, 0]
base_perf_train = calc_model_performance(y_train, y_pred_train)
base_perf_train
{'Root Mean Squared Error': 5964.030079525333,
'Mean Squared Error': 35569654.78948295,
'Mean Absolute Error': 4583.192074027712,
'Mean Absolute Percentage Error': 0.7487184929524596,
'R Squared': 0.7572131565075612}
base_perf_test = calc_model_performance(y_test, y_pred_test)
base_perf_test
{'Root Mean Squared Error': 5752.477398059859,
'Mean Squared Error': 33090996.215189524,
'Mean Absolute Error': 4534.422635060557,
'Mean Absolute Percentage Error': 0.7565371511851325,
'R Squared': 0.7741917232715523}
We can check the normality of residuals using a QQ (quantile-quantile) plot. This plots the value of each actual quantile (from the data) vs the theoretical quantile (assuming a normal distribution). If the data is perfectly normally distributed, you'd expect the datapoints to lie on the line.
We'll also use a histogram as a more interpretable visualisation of the residuals.
residuals_train = y_train - y_pred_train
residuals_test = y_test - y_pred_test
fig = sm.qqplot(
residuals_train,
fit=True,
line='45'
)
fig = sm.qqplot(
residuals_test,
fit=True,
line='45'
)
plot_residuals(y_true=y_train, y_pred=y_pred_train)
Our residuals are not perfectly normally distributed, but are pretty close!
We can check for homoscedasticity using a scatterplot, where the target is shown along the x-axis and the residuals are shown along the y-axis. We would expect the datapoints to be equally distributed across the y-axis as x (i.e. the target value) increases:
px.scatter(x=y_train, y=residuals_train)
px.scatter(x=y_test, y=residuals_test)
Our model exhibits significant heteroscedasticity (i.e. the variance of the residuals is not homogeneous with respect to the target). Since this is our baseline model
rfe = RFE(estimator=XGBRegressor())
xgb = XGBRegressor()
prep = ColumnTransformer([
('ohe', OneHotEncoder(), cat_cols)
],
remainder="passthrough"
)
pipe = Pipeline([
("preprocessor", prep),
("rfe", rfe),
("xgb", xgb)
])
Now we create our pipeline by specifying the list of sequential processes we want to run. The output of each step is passed to the next, with the final step being an estimator
num_features = X_train.shape[1]
search_spaces = {
'rfe__n_features_to_select': Integer(1, num_features), # Num features returned by RFE
'xgb__n_estimators': Integer(1, 500), # Num trees built by XGBoost
'xgb__max_depth': Integer(2, 8), # Max depth of trees built by XGBoost
'xgb__reg_lambda': Integer(1, 200), # Regularisation term (lambda) used in XGBoost
'xgb__learning_rate': Real(0, 1), # Learning rate used in XGBoost
'xgb__gamma': Real(0, 2000) # Gamma used in XGBoost
}
Now we can initiate the parameter optimisation process using our training set:
xgb_bs_cv = BayesSearchCV(
estimator=pipe, # Pipeline
search_spaces=search_spaces, # Search spaces
scoring='neg_root_mean_squared_error', # BayesSearchCV tries to maximise scoring metric, so negative RMSE used
n_iter=75, # Num of optimisation iterations
cv=3, # Number of folds
verbose=3, # Show progress
random_state=0 # Ensures reproducible results
)
xgb_bs_cv.fit(
X_train,
y_train,
)
Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=2, xgb__gamma=1352.1590177538271, xgb__learning_rate=0.622709752629437, xgb__max_depth=6, xgb__n_estimators=430, xgb__reg_lambda=196;, score=-6576.270 total time= 2.7s [CV 2/3] END rfe__n_features_to_select=2, xgb__gamma=1352.1590177538271, xgb__learning_rate=0.622709752629437, xgb__max_depth=6, xgb__n_estimators=430, xgb__reg_lambda=196;, score=-6982.957 total time= 1.4s [CV 3/3] END rfe__n_features_to_select=2, xgb__gamma=1352.1590177538271, xgb__learning_rate=0.622709752629437, xgb__max_depth=6, xgb__n_estimators=430, xgb__reg_lambda=196;, score=-6768.849 total time= 1.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=2, xgb__gamma=1957.8073168527233, xgb__learning_rate=0.9084383736699745, xgb__max_depth=6, xgb__n_estimators=132, xgb__reg_lambda=33;, score=-7091.444 total time= 1.0s [CV 2/3] END rfe__n_features_to_select=2, xgb__gamma=1957.8073168527233, xgb__learning_rate=0.9084383736699745, xgb__max_depth=6, xgb__n_estimators=132, xgb__reg_lambda=33;, score=-7158.632 total time= 0.6s [CV 3/3] END rfe__n_features_to_select=2, xgb__gamma=1957.8073168527233, xgb__learning_rate=0.9084383736699745, xgb__max_depth=6, xgb__n_estimators=132, xgb__reg_lambda=33;, score=-6970.892 total time= 1.0s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=2, xgb__gamma=115.7956873870605, xgb__learning_rate=0.3422834491646985, xgb__max_depth=3, xgb__n_estimators=121, xgb__reg_lambda=69;, score=-5882.660 total time= 1.1s [CV 2/3] END rfe__n_features_to_select=2, xgb__gamma=115.7956873870605, xgb__learning_rate=0.3422834491646985, xgb__max_depth=3, xgb__n_estimators=121, xgb__reg_lambda=69;, score=-6366.108 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=2, xgb__gamma=115.7956873870605, xgb__learning_rate=0.3422834491646985, xgb__max_depth=3, xgb__n_estimators=121, xgb__reg_lambda=69;, score=-6208.416 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=1, xgb__gamma=1566.5325413459877, xgb__learning_rate=0.46994404036291715, xgb__max_depth=5, xgb__n_estimators=409, xgb__reg_lambda=136;, score=-7472.190 total time= 0.8s [CV 2/3] END rfe__n_features_to_select=1, xgb__gamma=1566.5325413459877, xgb__learning_rate=0.46994404036291715, xgb__max_depth=5, xgb__n_estimators=409, xgb__reg_lambda=136;, score=-7831.296 total time= 2.0s [CV 3/3] END rfe__n_features_to_select=1, xgb__gamma=1566.5325413459877, xgb__learning_rate=0.46994404036291715, xgb__max_depth=5, xgb__n_estimators=409, xgb__reg_lambda=136;, score=-7537.046 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=1, xgb__gamma=486.671870936798, xgb__learning_rate=0.7281811891811247, xgb__max_depth=4, xgb__n_estimators=91, xgb__reg_lambda=51;, score=-7472.045 total time= 0.7s [CV 2/3] END rfe__n_features_to_select=1, xgb__gamma=486.671870936798, xgb__learning_rate=0.7281811891811247, xgb__max_depth=4, xgb__n_estimators=91, xgb__reg_lambda=51;, score=-7831.523 total time= 0.6s [CV 3/3] END rfe__n_features_to_select=1, xgb__gamma=486.671870936798, xgb__learning_rate=0.7281811891811247, xgb__max_depth=4, xgb__n_estimators=91, xgb__reg_lambda=51;, score=-7536.933 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1982.4481077857972, xgb__learning_rate=0.7334067033728838, xgb__max_depth=5, xgb__n_estimators=167, xgb__reg_lambda=57;, score=-4998.599 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1982.4481077857972, xgb__learning_rate=0.7334067033728838, xgb__max_depth=5, xgb__n_estimators=167, xgb__reg_lambda=57;, score=-5738.940 total time= 0.7s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1982.4481077857972, xgb__learning_rate=0.7334067033728838, xgb__max_depth=5, xgb__n_estimators=167, xgb__reg_lambda=57;, score=-5508.306 total time= 0.4s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1513.1058120840992, xgb__learning_rate=0.426882486653522, xgb__max_depth=3, xgb__n_estimators=120, xgb__reg_lambda=171;, score=-4220.442 total time= 0.7s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1513.1058120840992, xgb__learning_rate=0.426882486653522, xgb__max_depth=3, xgb__n_estimators=120, xgb__reg_lambda=171;, score=-4975.147 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1513.1058120840992, xgb__learning_rate=0.426882486653522, xgb__max_depth=3, xgb__n_estimators=120, xgb__reg_lambda=171;, score=-4956.275 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=2, xgb__gamma=1053.9580626300371, xgb__learning_rate=0.7217773579860347, xgb__max_depth=2, xgb__n_estimators=27, xgb__reg_lambda=49;, score=-5703.759 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=2, xgb__gamma=1053.9580626300371, xgb__learning_rate=0.7217773579860347, xgb__max_depth=2, xgb__n_estimators=27, xgb__reg_lambda=49;, score=-6298.026 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=2, xgb__gamma=1053.9580626300371, xgb__learning_rate=0.7217773579860347, xgb__max_depth=2, xgb__n_estimators=27, xgb__reg_lambda=49;, score=-6066.115 total time= 0.6s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=1, xgb__gamma=1792.3452357173578, xgb__learning_rate=0.7934820460664541, xgb__max_depth=8, xgb__n_estimators=225, xgb__reg_lambda=109;, score=-7472.147 total time= 0.6s [CV 2/3] END rfe__n_features_to_select=1, xgb__gamma=1792.3452357173578, xgb__learning_rate=0.7934820460664541, xgb__max_depth=8, xgb__n_estimators=225, xgb__reg_lambda=109;, score=-7831.359 total time= 0.8s [CV 3/3] END rfe__n_features_to_select=1, xgb__gamma=1792.3452357173578, xgb__learning_rate=0.7934820460664541, xgb__max_depth=8, xgb__n_estimators=225, xgb__reg_lambda=109;, score=-7536.995 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=2, xgb__gamma=880.3672674778031, xgb__learning_rate=0.2617387748127596, xgb__max_depth=7, xgb__n_estimators=36, xgb__reg_lambda=149;, score=-5570.514 total time= 0.9s [CV 2/3] END rfe__n_features_to_select=2, xgb__gamma=880.3672674778031, xgb__learning_rate=0.2617387748127596, xgb__max_depth=7, xgb__n_estimators=36, xgb__reg_lambda=149;, score=-6256.016 total time= 0.7s [CV 3/3] END rfe__n_features_to_select=2, xgb__gamma=880.3672674778031, xgb__learning_rate=0.2617387748127596, xgb__max_depth=7, xgb__n_estimators=36, xgb__reg_lambda=149;, score=-6143.075 total time= 0.6s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=8, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-19016.357 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=8, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-17163.288 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=8, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-17897.135 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.42109331177651455, xgb__max_depth=8, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-16206.073 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.42109331177651455, xgb__max_depth=8, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-14199.460 total time= 0.6s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.42109331177651455, xgb__max_depth=8, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-15063.426 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1852.4380031881344, xgb__learning_rate=0.8008835487383782, xgb__max_depth=4, xgb__n_estimators=139, xgb__reg_lambda=36;, score=-4953.037 total time= 0.6s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1852.4380031881344, xgb__learning_rate=0.8008835487383782, xgb__max_depth=4, xgb__n_estimators=139, xgb__reg_lambda=36;, score=-5605.223 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1852.4380031881344, xgb__learning_rate=0.8008835487383782, xgb__max_depth=4, xgb__n_estimators=139, xgb__reg_lambda=36;, score=-5382.842 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1501.6540526237834, xgb__learning_rate=0.4133538151471223, xgb__max_depth=3, xgb__n_estimators=80, xgb__reg_lambda=180;, score=-4157.400 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1501.6540526237834, xgb__learning_rate=0.4133538151471223, xgb__max_depth=3, xgb__n_estimators=80, xgb__reg_lambda=180;, score=-4965.766 total time= 0.6s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1501.6540526237834, xgb__learning_rate=0.4133538151471223, xgb__max_depth=3, xgb__n_estimators=80, xgb__reg_lambda=180;, score=-4902.953 total time= 0.4s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1457.2721059458133, xgb__learning_rate=0.35855682334810735, xgb__max_depth=3, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-16589.118 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1457.2721059458133, xgb__learning_rate=0.35855682334810735, xgb__max_depth=3, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-14598.675 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1457.2721059458133, xgb__learning_rate=0.35855682334810735, xgb__max_depth=3, xgb__n_estimators=1, xgb__reg_lambda=200;, score=-15441.840 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1195.958111754106, xgb__learning_rate=0.42292064755065356, xgb__max_depth=3, xgb__n_estimators=82, xgb__reg_lambda=179;, score=-4149.589 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1195.958111754106, xgb__learning_rate=0.42292064755065356, xgb__max_depth=3, xgb__n_estimators=82, xgb__reg_lambda=179;, score=-4962.582 total time= 0.7s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1195.958111754106, xgb__learning_rate=0.42292064755065356, xgb__max_depth=3, xgb__n_estimators=82, xgb__reg_lambda=179;, score=-4921.568 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=2, xgb__gamma=1120.4424328885364, xgb__learning_rate=0.6909829363217203, xgb__max_depth=2, xgb__n_estimators=45, xgb__reg_lambda=81;, score=-5678.470 total time= 0.5s [CV 2/3] END rfe__n_features_to_select=2, xgb__gamma=1120.4424328885364, xgb__learning_rate=0.6909829363217203, xgb__max_depth=2, xgb__n_estimators=45, xgb__reg_lambda=81;, score=-6245.905 total time= 0.8s [CV 3/3] END rfe__n_features_to_select=2, xgb__gamma=1120.4424328885364, xgb__learning_rate=0.6909829363217203, xgb__max_depth=2, xgb__n_estimators=45, xgb__reg_lambda=81;, score=-6087.424 total time= 0.9s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1383.505219247714, xgb__learning_rate=0.429139105981533, xgb__max_depth=2, xgb__n_estimators=83, xgb__reg_lambda=181;, score=-4222.400 total time= 0.7s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1383.505219247714, xgb__learning_rate=0.429139105981533, xgb__max_depth=2, xgb__n_estimators=83, xgb__reg_lambda=181;, score=-4988.136 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1383.505219247714, xgb__learning_rate=0.429139105981533, xgb__max_depth=2, xgb__n_estimators=83, xgb__reg_lambda=181;, score=-4883.128 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=0.0, xgb__learning_rate=0.2880536442863135, xgb__max_depth=8, xgb__n_estimators=120, xgb__reg_lambda=174;, score=-4376.112 total time= 1.0s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=0.0, xgb__learning_rate=0.2880536442863135, xgb__max_depth=8, xgb__n_estimators=120, xgb__reg_lambda=174;, score=-5134.587 total time= 0.6s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=0.0, xgb__learning_rate=0.2880536442863135, xgb__max_depth=8, xgb__n_estimators=120, xgb__reg_lambda=174;, score=-5077.969 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=150.00305575841324, xgb__learning_rate=0.425243994944344, xgb__max_depth=7, xgb__n_estimators=80, xgb__reg_lambda=151;, score=-4368.686 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=150.00305575841324, xgb__learning_rate=0.425243994944344, xgb__max_depth=7, xgb__n_estimators=80, xgb__reg_lambda=151;, score=-5109.547 total time= 0.7s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=150.00305575841324, xgb__learning_rate=0.425243994944344, xgb__max_depth=7, xgb__n_estimators=80, xgb__reg_lambda=151;, score=-5094.375 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1743.6763377080008, xgb__learning_rate=0.4069857342107497, xgb__max_depth=2, xgb__n_estimators=81, xgb__reg_lambda=200;, score=-4244.940 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1743.6763377080008, xgb__learning_rate=0.4069857342107497, xgb__max_depth=2, xgb__n_estimators=81, xgb__reg_lambda=200;, score=-4984.745 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1743.6763377080008, xgb__learning_rate=0.4069857342107497, xgb__max_depth=2, xgb__n_estimators=81, xgb__reg_lambda=200;, score=-4883.309 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.43382645014819426, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=67;, score=-4259.115 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.43382645014819426, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=67;, score=-4976.225 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.43382645014819426, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=67;, score=-4952.638 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=848.0675373990235, xgb__learning_rate=0.4588597548442855, xgb__max_depth=2, xgb__n_estimators=80, xgb__reg_lambda=194;, score=-4200.238 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=848.0675373990235, xgb__learning_rate=0.4588597548442855, xgb__max_depth=2, xgb__n_estimators=80, xgb__reg_lambda=194;, score=-4998.266 total time= 0.6s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=848.0675373990235, xgb__learning_rate=0.4588597548442855, xgb__max_depth=2, xgb__n_estimators=80, xgb__reg_lambda=194;, score=-4903.144 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1225.1610608027681, xgb__learning_rate=0.35085070634064325, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=200;, score=-4202.813 total time= 0.5s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1225.1610608027681, xgb__learning_rate=0.35085070634064325, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=200;, score=-4968.275 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1225.1610608027681, xgb__learning_rate=0.35085070634064325, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=200;, score=-4906.176 total time= 0.4s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1872.1389302908663, xgb__learning_rate=0.4236029761301352, xgb__max_depth=2, xgb__n_estimators=116, xgb__reg_lambda=200;, score=-4268.801 total time= 0.9s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1872.1389302908663, xgb__learning_rate=0.4236029761301352, xgb__max_depth=2, xgb__n_estimators=116, xgb__reg_lambda=200;, score=-4999.901 total time= 0.7s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1872.1389302908663, xgb__learning_rate=0.4236029761301352, xgb__max_depth=2, xgb__n_estimators=116, xgb__reg_lambda=200;, score=-4896.797 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.36297945610082927, xgb__max_depth=2, xgb__n_estimators=81, xgb__reg_lambda=200;, score=-4228.462 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.36297945610082927, xgb__max_depth=2, xgb__n_estimators=81, xgb__reg_lambda=200;, score=-4982.571 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.36297945610082927, xgb__max_depth=2, xgb__n_estimators=81, xgb__reg_lambda=200;, score=-4898.347 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1320.945172169262, xgb__learning_rate=0.4101985856880926, xgb__max_depth=2, xgb__n_estimators=121, xgb__reg_lambda=200;, score=-4226.744 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1320.945172169262, xgb__learning_rate=0.4101985856880926, xgb__max_depth=2, xgb__n_estimators=121, xgb__reg_lambda=200;, score=-4986.349 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1320.945172169262, xgb__learning_rate=0.4101985856880926, xgb__max_depth=2, xgb__n_estimators=121, xgb__reg_lambda=200;, score=-4889.849 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=692.0423224467922, xgb__learning_rate=0.4072354991973613, xgb__max_depth=2, xgb__n_estimators=122, xgb__reg_lambda=193;, score=-4249.029 total time= 0.5s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=692.0423224467922, xgb__learning_rate=0.4072354991973613, xgb__max_depth=2, xgb__n_estimators=122, xgb__reg_lambda=193;, score=-4990.556 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=692.0423224467922, xgb__learning_rate=0.4072354991973613, xgb__max_depth=2, xgb__n_estimators=122, xgb__reg_lambda=193;, score=-4892.552 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.4214871139577795, xgb__max_depth=2, xgb__n_estimators=82, xgb__reg_lambda=197;, score=-4202.755 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.4214871139577795, xgb__max_depth=2, xgb__n_estimators=82, xgb__reg_lambda=197;, score=-4982.177 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.4214871139577795, xgb__max_depth=2, xgb__n_estimators=82, xgb__reg_lambda=197;, score=-4880.227 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=2, xgb__n_estimators=196, xgb__reg_lambda=200;, score=-19016.357 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=2, xgb__n_estimators=196, xgb__reg_lambda=200;, score=-17163.288 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=2, xgb__n_estimators=196, xgb__reg_lambda=200;, score=-17897.135 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=1, xgb__gamma=1600.7200535199743, xgb__learning_rate=0.38503768155952856, xgb__max_depth=2, xgb__n_estimators=33, xgb__reg_lambda=200;, score=-7481.479 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=1, xgb__gamma=1600.7200535199743, xgb__learning_rate=0.38503768155952856, xgb__max_depth=2, xgb__n_estimators=33, xgb__reg_lambda=200;, score=-7817.753 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=1, xgb__gamma=1600.7200535199743, xgb__learning_rate=0.38503768155952856, xgb__max_depth=2, xgb__n_estimators=33, xgb__reg_lambda=200;, score=-7544.334 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=277.3108385275214, xgb__learning_rate=0.35509644257114303, xgb__max_depth=2, xgb__n_estimators=43, xgb__reg_lambda=198;, score=-4230.027 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=277.3108385275214, xgb__learning_rate=0.35509644257114303, xgb__max_depth=2, xgb__n_estimators=43, xgb__reg_lambda=198;, score=-5013.004 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=277.3108385275214, xgb__learning_rate=0.35509644257114303, xgb__max_depth=2, xgb__n_estimators=43, xgb__reg_lambda=198;, score=-4977.744 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=163, xgb__reg_lambda=197;, score=-4280.675 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=163, xgb__reg_lambda=197;, score=-5135.199 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=163, xgb__reg_lambda=197;, score=-4957.659 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1822.1991338677321, xgb__learning_rate=0.7382220160706185, xgb__max_depth=2, xgb__n_estimators=37, xgb__reg_lambda=156;, score=-4180.527 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1822.1991338677321, xgb__learning_rate=0.7382220160706185, xgb__max_depth=2, xgb__n_estimators=37, xgb__reg_lambda=156;, score=-5008.444 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1822.1991338677321, xgb__learning_rate=0.7382220160706185, xgb__max_depth=2, xgb__n_estimators=37, xgb__reg_lambda=156;, score=-4927.630 total time= 0.6s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1327.1170090955811, xgb__learning_rate=0.9413575523394221, xgb__max_depth=2, xgb__n_estimators=34, xgb__reg_lambda=54;, score=-4164.874 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1327.1170090955811, xgb__learning_rate=0.9413575523394221, xgb__max_depth=2, xgb__n_estimators=34, xgb__reg_lambda=54;, score=-5018.235 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1327.1170090955811, xgb__learning_rate=0.9413575523394221, xgb__max_depth=2, xgb__n_estimators=34, xgb__reg_lambda=54;, score=-4890.614 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=1, xgb__gamma=648.9036078481012, xgb__learning_rate=1.0, xgb__max_depth=8, xgb__n_estimators=257, xgb__reg_lambda=1;, score=-7472.071 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=1, xgb__gamma=648.9036078481012, xgb__learning_rate=1.0, xgb__max_depth=8, xgb__n_estimators=257, xgb__reg_lambda=1;, score=-7831.555 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=1, xgb__gamma=648.9036078481012, xgb__learning_rate=1.0, xgb__max_depth=8, xgb__n_estimators=257, xgb__reg_lambda=1;, score=-7536.923 total time= 0.6s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1384.8853758842768, xgb__learning_rate=1.0, xgb__max_depth=7, xgb__n_estimators=500, xgb__reg_lambda=187;, score=-5734.483 total time= 1.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1384.8853758842768, xgb__learning_rate=1.0, xgb__max_depth=7, xgb__n_estimators=500, xgb__reg_lambda=187;, score=-6081.685 total time= 1.6s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1384.8853758842768, xgb__learning_rate=1.0, xgb__max_depth=7, xgb__n_estimators=500, xgb__reg_lambda=187;, score=-5840.018 total time= 1.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.07889275937014419, xgb__max_depth=2, xgb__n_estimators=53, xgb__reg_lambda=200;, score=-7284.785 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.07889275937014419, xgb__max_depth=2, xgb__n_estimators=53, xgb__reg_lambda=200;, score=-6560.995 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.07889275937014419, xgb__max_depth=2, xgb__n_estimators=53, xgb__reg_lambda=200;, score=-7169.582 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=124.44661412227939, xgb__learning_rate=0.44228486682581286, xgb__max_depth=2, xgb__n_estimators=41, xgb__reg_lambda=194;, score=-4223.673 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=124.44661412227939, xgb__learning_rate=0.44228486682581286, xgb__max_depth=2, xgb__n_estimators=41, xgb__reg_lambda=194;, score=-4997.960 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=124.44661412227939, xgb__learning_rate=0.44228486682581286, xgb__max_depth=2, xgb__n_estimators=41, xgb__reg_lambda=194;, score=-4911.533 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.8299318246247381, xgb__max_depth=2, xgb__n_estimators=158, xgb__reg_lambda=99;, score=-4357.253 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.8299318246247381, xgb__max_depth=2, xgb__n_estimators=158, xgb__reg_lambda=99;, score=-5217.995 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.8299318246247381, xgb__max_depth=2, xgb__n_estimators=158, xgb__reg_lambda=99;, score=-5075.831 total time= 0.4s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.8691583470952974, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=200;, score=-4199.961 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.8691583470952974, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=200;, score=-5023.885 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.8691583470952974, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=200;, score=-4947.263 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1310.701006659733, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=105;, score=-4245.644 total time= 0.6s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1310.701006659733, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=105;, score=-5152.636 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1310.701006659733, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=105;, score=-5019.362 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1581.7185465629445, xgb__learning_rate=0.8698501481995106, xgb__max_depth=2, xgb__n_estimators=33, xgb__reg_lambda=1;, score=-4546.010 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1581.7185465629445, xgb__learning_rate=0.8698501481995106, xgb__max_depth=2, xgb__n_estimators=33, xgb__reg_lambda=1;, score=-5246.911 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1581.7185465629445, xgb__learning_rate=0.8698501481995106, xgb__max_depth=2, xgb__n_estimators=33, xgb__reg_lambda=1;, score=-5187.867 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1498.889459670576, xgb__learning_rate=0.7603014300951674, xgb__max_depth=2, xgb__n_estimators=41, xgb__reg_lambda=200;, score=-4229.825 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1498.889459670576, xgb__learning_rate=0.7603014300951674, xgb__max_depth=2, xgb__n_estimators=41, xgb__reg_lambda=200;, score=-5027.241 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1498.889459670576, xgb__learning_rate=0.7603014300951674, xgb__max_depth=2, xgb__n_estimators=41, xgb__reg_lambda=200;, score=-4900.755 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.5811857774472619, xgb__max_depth=2, xgb__n_estimators=120, xgb__reg_lambda=200;, score=-4278.880 total time= 0.5s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.5811857774472619, xgb__max_depth=2, xgb__n_estimators=120, xgb__reg_lambda=200;, score=-4985.978 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.5811857774472619, xgb__max_depth=2, xgb__n_estimators=120, xgb__reg_lambda=200;, score=-4910.856 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=881.5450284325716, xgb__learning_rate=0.614511381947848, xgb__max_depth=2, xgb__n_estimators=40, xgb__reg_lambda=200;, score=-4270.201 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=881.5450284325716, xgb__learning_rate=0.614511381947848, xgb__max_depth=2, xgb__n_estimators=40, xgb__reg_lambda=200;, score=-5022.935 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=881.5450284325716, xgb__learning_rate=0.614511381947848, xgb__max_depth=2, xgb__n_estimators=40, xgb__reg_lambda=200;, score=-4894.314 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=2, xgb__n_estimators=135, xgb__reg_lambda=163;, score=-19016.357 total time= 0.8s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=2, xgb__n_estimators=135, xgb__reg_lambda=163;, score=-17163.288 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.0, xgb__max_depth=2, xgb__n_estimators=135, xgb__reg_lambda=163;, score=-17897.135 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=324.8759595744798, xgb__learning_rate=0.44177649551803233, xgb__max_depth=2, xgb__n_estimators=45, xgb__reg_lambda=200;, score=-4226.641 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=324.8759595744798, xgb__learning_rate=0.44177649551803233, xgb__max_depth=2, xgb__n_estimators=45, xgb__reg_lambda=200;, score=-4982.545 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=324.8759595744798, xgb__learning_rate=0.44177649551803233, xgb__max_depth=2, xgb__n_estimators=45, xgb__reg_lambda=200;, score=-4900.062 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1292.817428563956, xgb__learning_rate=0.9430456955472653, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=200;, score=-4197.058 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1292.817428563956, xgb__learning_rate=0.9430456955472653, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=200;, score=-5075.965 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1292.817428563956, xgb__learning_rate=0.9430456955472653, xgb__max_depth=2, xgb__n_estimators=35, xgb__reg_lambda=200;, score=-4900.460 total time= 0.4s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=431.11624731286753, xgb__learning_rate=0.340949197544821, xgb__max_depth=5, xgb__n_estimators=117, xgb__reg_lambda=200;, score=-4305.067 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=431.11624731286753, xgb__learning_rate=0.340949197544821, xgb__max_depth=5, xgb__n_estimators=117, xgb__reg_lambda=200;, score=-5054.303 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=431.11624731286753, xgb__learning_rate=0.340949197544821, xgb__max_depth=5, xgb__n_estimators=117, xgb__reg_lambda=200;, score=-4984.784 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=291.52512205032247, xgb__learning_rate=0.3436033243546065, xgb__max_depth=8, xgb__n_estimators=117, xgb__reg_lambda=190;, score=-4418.560 total time= 0.7s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=291.52512205032247, xgb__learning_rate=0.3436033243546065, xgb__max_depth=8, xgb__n_estimators=117, xgb__reg_lambda=190;, score=-5131.775 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=291.52512205032247, xgb__learning_rate=0.3436033243546065, xgb__max_depth=8, xgb__n_estimators=117, xgb__reg_lambda=190;, score=-5104.657 total time= 0.6s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1991.5316193074125, xgb__learning_rate=0.9319197339814749, xgb__max_depth=2, xgb__n_estimators=37, xgb__reg_lambda=200;, score=-4177.758 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1991.5316193074125, xgb__learning_rate=0.9319197339814749, xgb__max_depth=2, xgb__n_estimators=37, xgb__reg_lambda=200;, score=-5063.916 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1991.5316193074125, xgb__learning_rate=0.9319197339814749, xgb__max_depth=2, xgb__n_estimators=37, xgb__reg_lambda=200;, score=-4949.611 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.4872845601568649, xgb__max_depth=2, xgb__n_estimators=121, xgb__reg_lambda=200;, score=-4238.886 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.4872845601568649, xgb__max_depth=2, xgb__n_estimators=121, xgb__reg_lambda=200;, score=-4988.330 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.4872845601568649, xgb__max_depth=2, xgb__n_estimators=121, xgb__reg_lambda=200;, score=-4922.528 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.9809129692826738, xgb__max_depth=2, xgb__n_estimators=171, xgb__reg_lambda=200;, score=-4308.820 total time= 0.6s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.9809129692826738, xgb__max_depth=2, xgb__n_estimators=171, xgb__reg_lambda=200;, score=-5131.962 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.9809129692826738, xgb__max_depth=2, xgb__n_estimators=171, xgb__reg_lambda=200;, score=-4965.681 total time= 1.0s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.5576492861523851, xgb__max_depth=2, xgb__n_estimators=105, xgb__reg_lambda=200;, score=-4279.225 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.5576492861523851, xgb__max_depth=2, xgb__n_estimators=105, xgb__reg_lambda=200;, score=-4965.990 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.5576492861523851, xgb__max_depth=2, xgb__n_estimators=105, xgb__reg_lambda=200;, score=-4935.300 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1173.3646157784028, xgb__learning_rate=0.32231892235796333, xgb__max_depth=2, xgb__n_estimators=48, xgb__reg_lambda=200;, score=-4241.064 total time= 0.6s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1173.3646157784028, xgb__learning_rate=0.32231892235796333, xgb__max_depth=2, xgb__n_estimators=48, xgb__reg_lambda=200;, score=-4992.731 total time= 0.8s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1173.3646157784028, xgb__learning_rate=0.32231892235796333, xgb__max_depth=2, xgb__n_estimators=48, xgb__reg_lambda=200;, score=-4975.289 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1258.43530598402, xgb__learning_rate=0.9680906002855667, xgb__max_depth=2, xgb__n_estimators=32, xgb__reg_lambda=200;, score=-4252.158 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1258.43530598402, xgb__learning_rate=0.9680906002855667, xgb__max_depth=2, xgb__n_estimators=32, xgb__reg_lambda=200;, score=-5080.001 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1258.43530598402, xgb__learning_rate=0.9680906002855667, xgb__max_depth=2, xgb__n_estimators=32, xgb__reg_lambda=200;, score=-4907.887 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=846.1164117185391, xgb__learning_rate=0.0, xgb__max_depth=7, xgb__n_estimators=500, xgb__reg_lambda=84;, score=-19016.357 total time= 0.5s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=846.1164117185391, xgb__learning_rate=0.0, xgb__max_depth=7, xgb__n_estimators=500, xgb__reg_lambda=84;, score=-17163.288 total time= 1.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=846.1164117185391, xgb__learning_rate=0.0, xgb__max_depth=7, xgb__n_estimators=500, xgb__reg_lambda=84;, score=-17897.135 total time= 2.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=2, xgb__gamma=1637.9290634447893, xgb__learning_rate=1.0, xgb__max_depth=5, xgb__n_estimators=430, xgb__reg_lambda=200;, score=-6708.458 total time= 1.1s [CV 2/3] END rfe__n_features_to_select=2, xgb__gamma=1637.9290634447893, xgb__learning_rate=1.0, xgb__max_depth=5, xgb__n_estimators=430, xgb__reg_lambda=200;, score=-7012.815 total time= 1.3s [CV 3/3] END rfe__n_features_to_select=2, xgb__gamma=1637.9290634447893, xgb__learning_rate=1.0, xgb__max_depth=5, xgb__n_estimators=430, xgb__reg_lambda=200;, score=-6794.054 total time= 1.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=0.0, xgb__learning_rate=0.3146852676749732, xgb__max_depth=6, xgb__n_estimators=48, xgb__reg_lambda=200;, score=-4166.729 total time= 0.7s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=0.0, xgb__learning_rate=0.3146852676749732, xgb__max_depth=6, xgb__n_estimators=48, xgb__reg_lambda=200;, score=-5008.250 total time= 0.6s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=0.0, xgb__learning_rate=0.3146852676749732, xgb__max_depth=6, xgb__n_estimators=48, xgb__reg_lambda=200;, score=-4993.017 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.3528745194156683, xgb__max_depth=2, xgb__n_estimators=123, xgb__reg_lambda=200;, score=-4219.992 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.3528745194156683, xgb__max_depth=2, xgb__n_estimators=123, xgb__reg_lambda=200;, score=-4977.579 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.3528745194156683, xgb__max_depth=2, xgb__n_estimators=123, xgb__reg_lambda=200;, score=-4911.460 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1193.8405631543962, xgb__learning_rate=0.3994465580067442, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=200;, score=-4201.098 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1193.8405631543962, xgb__learning_rate=0.3994465580067442, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=200;, score=-4998.572 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1193.8405631543962, xgb__learning_rate=0.3994465580067442, xgb__max_depth=2, xgb__n_estimators=84, xgb__reg_lambda=200;, score=-4879.455 total time= 0.6s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1798.2586451592604, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=348, xgb__reg_lambda=200;, score=-4345.472 total time= 0.5s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1798.2586451592604, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=348, xgb__reg_lambda=200;, score=-5203.061 total time= 0.7s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1798.2586451592604, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=348, xgb__reg_lambda=200;, score=-5063.940 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1075.7938547023703, xgb__learning_rate=0.8557314423502477, xgb__max_depth=2, xgb__n_estimators=349, xgb__reg_lambda=200;, score=-4369.841 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1075.7938547023703, xgb__learning_rate=0.8557314423502477, xgb__max_depth=2, xgb__n_estimators=349, xgb__reg_lambda=200;, score=-5143.437 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1075.7938547023703, xgb__learning_rate=0.8557314423502477, xgb__max_depth=2, xgb__n_estimators=349, xgb__reg_lambda=200;, score=-5070.949 total time= 0.6s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=7.815651348225705, xgb__learning_rate=0.9168672356435439, xgb__max_depth=8, xgb__n_estimators=362, xgb__reg_lambda=183;, score=-5535.969 total time= 0.7s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=7.815651348225705, xgb__learning_rate=0.9168672356435439, xgb__max_depth=8, xgb__n_estimators=362, xgb__reg_lambda=183;, score=-5884.535 total time= 0.8s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=7.815651348225705, xgb__learning_rate=0.9168672356435439, xgb__max_depth=8, xgb__n_estimators=362, xgb__reg_lambda=183;, score=-5728.355 total time= 0.7s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.898763159989001, xgb__max_depth=2, xgb__n_estimators=338, xgb__reg_lambda=191;, score=-4347.182 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.898763159989001, xgb__max_depth=2, xgb__n_estimators=338, xgb__reg_lambda=191;, score=-5192.998 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.898763159989001, xgb__max_depth=2, xgb__n_estimators=338, xgb__reg_lambda=191;, score=-5057.108 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.7119875024090674, xgb__max_depth=2, xgb__n_estimators=344, xgb__reg_lambda=57;, score=-4487.986 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.7119875024090674, xgb__max_depth=2, xgb__n_estimators=344, xgb__reg_lambda=57;, score=-5238.644 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.7119875024090674, xgb__max_depth=2, xgb__n_estimators=344, xgb__reg_lambda=57;, score=-5128.790 total time= 0.4s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1903.9970991357882, xgb__learning_rate=0.3176958708749317, xgb__max_depth=2, xgb__n_estimators=118, xgb__reg_lambda=71;, score=-4220.976 total time= 0.4s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1903.9970991357882, xgb__learning_rate=0.3176958708749317, xgb__max_depth=2, xgb__n_estimators=118, xgb__reg_lambda=71;, score=-4978.594 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1903.9970991357882, xgb__learning_rate=0.3176958708749317, xgb__max_depth=2, xgb__n_estimators=118, xgb__reg_lambda=71;, score=-4925.087 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=1, xgb__gamma=1723.132937584645, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=341, xgb__reg_lambda=88;, score=-7472.103 total time= 0.6s [CV 2/3] END rfe__n_features_to_select=1, xgb__gamma=1723.132937584645, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=341, xgb__reg_lambda=88;, score=-7831.368 total time= 0.7s [CV 3/3] END rfe__n_features_to_select=1, xgb__gamma=1723.132937584645, xgb__learning_rate=1.0, xgb__max_depth=2, xgb__n_estimators=341, xgb__reg_lambda=88;, score=-7536.998 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=887.1621838129954, xgb__learning_rate=0.7105199870542571, xgb__max_depth=2, xgb__n_estimators=39, xgb__reg_lambda=200;, score=-4224.912 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=887.1621838129954, xgb__learning_rate=0.7105199870542571, xgb__max_depth=2, xgb__n_estimators=39, xgb__reg_lambda=200;, score=-5005.650 total time= 0.3s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=887.1621838129954, xgb__learning_rate=0.7105199870542571, xgb__max_depth=2, xgb__n_estimators=39, xgb__reg_lambda=200;, score=-4918.457 total time= 0.2s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1442.0862189000577, xgb__learning_rate=0.49822732234012357, xgb__max_depth=2, xgb__n_estimators=44, xgb__reg_lambda=200;, score=-4302.356 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1442.0862189000577, xgb__learning_rate=0.49822732234012357, xgb__max_depth=2, xgb__n_estimators=44, xgb__reg_lambda=200;, score=-5018.964 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1442.0862189000577, xgb__learning_rate=0.49822732234012357, xgb__max_depth=2, xgb__n_estimators=44, xgb__reg_lambda=200;, score=-4878.717 total time= 0.1s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.43545532271013737, xgb__max_depth=2, xgb__n_estimators=83, xgb__reg_lambda=200;, score=-4193.801 total time= 0.2s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.43545532271013737, xgb__max_depth=2, xgb__n_estimators=83, xgb__reg_lambda=200;, score=-4964.926 total time= 0.2s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=2000.0, xgb__learning_rate=0.43545532271013737, xgb__max_depth=2, xgb__n_estimators=83, xgb__reg_lambda=200;, score=-4887.719 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=797.6206961721583, xgb__learning_rate=0.22020626907860252, xgb__max_depth=2, xgb__n_estimators=339, xgb__reg_lambda=200;, score=-4279.409 total time= 0.8s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=797.6206961721583, xgb__learning_rate=0.22020626907860252, xgb__max_depth=2, xgb__n_estimators=339, xgb__reg_lambda=200;, score=-4992.283 total time= 0.8s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=797.6206961721583, xgb__learning_rate=0.22020626907860252, xgb__max_depth=2, xgb__n_estimators=339, xgb__reg_lambda=200;, score=-4906.522 total time= 0.5s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=1360.7433507629464, xgb__learning_rate=0.12272070725394739, xgb__max_depth=2, xgb__n_estimators=339, xgb__reg_lambda=189;, score=-4197.804 total time= 0.5s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=1360.7433507629464, xgb__learning_rate=0.12272070725394739, xgb__max_depth=2, xgb__n_estimators=339, xgb__reg_lambda=189;, score=-4963.191 total time= 0.4s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=1360.7433507629464, xgb__learning_rate=0.12272070725394739, xgb__max_depth=2, xgb__n_estimators=339, xgb__reg_lambda=189;, score=-4886.382 total time= 0.3s Fitting 3 folds for each of 1 candidates, totalling 3 fits [CV 1/3] END rfe__n_features_to_select=3, xgb__gamma=905.1460912518813, xgb__learning_rate=0.1258177257463098, xgb__max_depth=2, xgb__n_estimators=336, xgb__reg_lambda=200;, score=-4202.986 total time= 0.3s [CV 2/3] END rfe__n_features_to_select=3, xgb__gamma=905.1460912518813, xgb__learning_rate=0.1258177257463098, xgb__max_depth=2, xgb__n_estimators=336, xgb__reg_lambda=200;, score=-4973.328 total time= 0.5s [CV 3/3] END rfe__n_features_to_select=3, xgb__gamma=905.1460912518813, xgb__learning_rate=0.1258177257463098, xgb__max_depth=2, xgb__n_estimators=336, xgb__reg_lambda=200;, score=-4885.647 total time= 0.3s
BayesSearchCV(cv=3,
estimator=Pipeline(steps=[('preprocessor',
ColumnTransformer(remainder='passthrough',
transformers=[('ohe',
OneHotEncoder(),
['smoker'])])),
('rfe',
RFE(estimator=XGBRegressor(base_score=None,
booster=None,
callbacks=None,
colsample_bylevel=None,
colsample_bynode=None,
colsample_bytree=None,
early_stopping_rounds=None,
enable_categorical=Fals...
'xgb__gamma': Real(low=0, high=2000, prior='uniform', transform='normalize'),
'xgb__learning_rate': Real(low=0, high=1, prior='uniform', transform='normalize'),
'xgb__max_depth': Integer(low=2, high=8, prior='uniform', transform='normalize'),
'xgb__n_estimators': Integer(low=1, high=500, prior='uniform', transform='normalize'),
'xgb__reg_lambda': Integer(low=1, high=200, prior='uniform', transform='normalize')},
verbose=3)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. BayesSearchCV(cv=3,
estimator=Pipeline(steps=[('preprocessor',
ColumnTransformer(remainder='passthrough',
transformers=[('ohe',
OneHotEncoder(),
['smoker'])])),
('rfe',
RFE(estimator=XGBRegressor(base_score=None,
booster=None,
callbacks=None,
colsample_bylevel=None,
colsample_bynode=None,
colsample_bytree=None,
early_stopping_rounds=None,
enable_categorical=Fals...
'xgb__gamma': Real(low=0, high=2000, prior='uniform', transform='normalize'),
'xgb__learning_rate': Real(low=0, high=1, prior='uniform', transform='normalize'),
'xgb__max_depth': Integer(low=2, high=8, prior='uniform', transform='normalize'),
'xgb__n_estimators': Integer(low=1, high=500, prior='uniform', transform='normalize'),
'xgb__reg_lambda': Integer(low=1, high=200, prior='uniform', transform='normalize')},
verbose=3)Pipeline(steps=[('preprocessor',
ColumnTransformer(remainder='passthrough',
transformers=[('ohe', OneHotEncoder(),
['smoker'])])),
('rfe',
RFE(estimator=XGBRegressor(base_score=None, booster=None,
callbacks=None,
colsample_bylevel=None,
colsample_bynode=None,
colsample_bytree=None,
early_stopping_rounds=None,
enable_categorical=False,
eval_metric=None,
feature_ty...
feature_types=None, gamma=None, gpu_id=None,
grow_policy=None, importance_type=None,
interaction_constraints=None, learning_rate=None,
max_bin=None, max_cat_threshold=None,
max_cat_to_onehot=None, max_delta_step=None,
max_depth=None, max_leaves=None,
min_child_weight=None, missing=nan,
monotone_constraints=None, n_estimators=100,
n_jobs=None, num_parallel_tree=None,
predictor=None, random_state=None, ...))])ColumnTransformer(remainder='passthrough',
transformers=[('ohe', OneHotEncoder(), ['smoker'])])['smoker']
OneHotEncoder()
passthrough
RFE(estimator=XGBRegressor(base_score=None, booster=None, callbacks=None,
colsample_bylevel=None, colsample_bynode=None,
colsample_bytree=None, early_stopping_rounds=None,
enable_categorical=False, eval_metric=None,
feature_types=None, gamma=None, gpu_id=None,
grow_policy=None, importance_type=None,
interaction_constraints=None, learning_rate=None,
max_bin=None, max_cat_threshold=None,
max_cat_to_onehot=None, max_delta_step=None,
max_depth=None, max_leaves=None,
min_child_weight=None, missing=nan,
monotone_constraints=None, n_estimators=100,
n_jobs=None, num_parallel_tree=None, predictor=None,
random_state=None, ...))XGBRegressor(base_score=None, booster=None, callbacks=None,
colsample_bylevel=None, colsample_bynode=None,
colsample_bytree=None, early_stopping_rounds=None,
enable_categorical=False, eval_metric=None, feature_types=None,
gamma=None, gpu_id=None, grow_policy=None, importance_type=None,
interaction_constraints=None, learning_rate=None, max_bin=None,
max_cat_threshold=None, max_cat_to_onehot=None,
max_delta_step=None, max_depth=None, max_leaves=None,
min_child_weight=None, missing=nan, monotone_constraints=None,
n_estimators=100, n_jobs=None, num_parallel_tree=None,
predictor=None, random_state=None, ...)XGBRegressor(base_score=None, booster=None, callbacks=None,
colsample_bylevel=None, colsample_bynode=None,
colsample_bytree=None, early_stopping_rounds=None,
enable_categorical=False, eval_metric=None, feature_types=None,
gamma=None, gpu_id=None, grow_policy=None, importance_type=None,
interaction_constraints=None, learning_rate=None, max_bin=None,
max_cat_threshold=None, max_cat_to_onehot=None,
max_delta_step=None, max_depth=None, max_leaves=None,
min_child_weight=None, missing=nan, monotone_constraints=None,
n_estimators=100, n_jobs=None, num_parallel_tree=None,
predictor=None, random_state=None, ...)XGBRegressor(base_score=None, booster=None, callbacks=None,
colsample_bylevel=None, colsample_bynode=None,
colsample_bytree=None, early_stopping_rounds=None,
enable_categorical=False, eval_metric=None, feature_types=None,
gamma=None, gpu_id=None, grow_policy=None, importance_type=None,
interaction_constraints=None, learning_rate=None, max_bin=None,
max_cat_threshold=None, max_cat_to_onehot=None,
max_delta_step=None, max_depth=None, max_leaves=None,
min_child_weight=None, missing=nan, monotone_constraints=None,
n_estimators=100, n_jobs=None, num_parallel_tree=None,
predictor=None, random_state=None, ...)Let's first look at how each parameter set performed across each fold. Each record in the dataset corresponds to a parameter set that was tested. We rank by rank_test_score to ensure the best performing parameter set is shown at the top:
cv_results = pd.DataFrame(xgb_bs_cv.cv_results_).sort_values('rank_test_score')
cv_results
| mean_fit_time | std_fit_time | mean_score_time | std_score_time | param_rfe__n_features_to_select | param_xgb__gamma | param_xgb__learning_rate | param_xgb__max_depth | param_xgb__n_estimators | param_xgb__reg_lambda | params | split0_test_score | split1_test_score | split2_test_score | mean_test_score | std_test_score | rank_test_score | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 13 | 0.492979 | 0.114396 | 0.012567 | 0.003726 | 3 | 1501.654053 | 0.413354 | 3 | 80 | 180 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -4157.400461 | -4965.765998 | -4902.952577 | -4675.373012 | 367.158507 | 1 |
| 15 | 0.501304 | 0.181447 | 0.007560 | 0.001548 | 3 | 1195.958112 | 0.422921 | 3 | 82 | 179 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -4149.589191 | -4962.581890 | -4921.567740 | -4677.912940 | 373.956350 | 2 |
| 71 | 0.312757 | 0.037059 | 0.006004 | 0.003571 | 3 | 2000.0 | 0.435455 | 2 | 83 | 200 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -4193.800802 | -4964.925935 | -4887.719355 | -4682.148697 | 346.749635 | 3 |
| 73 | 0.497856 | 0.058181 | 0.006068 | 0.001244 | 3 | 1360.743351 | 0.122721 | 2 | 339 | 189 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -4197.804409 | -4963.190934 | -4886.382293 | -4682.459212 | 344.134277 | 4 |
| 74 | 0.455696 | 0.069277 | 0.010096 | 0.002348 | 3 | 905.146091 | 0.125818 | 2 | 336 | 200 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -4202.985731 | -4973.328374 | -4885.647110 | -4687.320405 | 344.341941 | 5 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 14 | 0.427442 | 0.235628 | 0.010019 | 0.003919 | 3 | 1457.272106 | 0.358557 | 3 | 1 | 200 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -16589.117726 | -14598.674803 | -15441.840268 | -15543.210932 | 815.750275 | 71 |
| 57 | 1.557921 | 0.799276 | 0.015366 | 0.012994 | 3 | 846.116412 | 0.0 | 7 | 500 | 84 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -19016.356974 | -17163.287836 | -17897.134903 | -18025.593238 | 761.945958 | 72 |
| 46 | 0.591750 | 0.227291 | 0.010546 | 0.004362 | 3 | 2000.0 | 0.0 | 2 | 135 | 163 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -19016.356974 | -17163.287836 | -17897.134903 | -18025.593238 | 761.945958 | 72 |
| 29 | 0.434777 | 0.074577 | 0.010565 | 0.002908 | 3 | 2000.0 | 0.0 | 2 | 196 | 200 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -19016.356974 | -17163.287836 | -17897.134903 | -18025.593238 | 761.945958 | 72 |
| 10 | 0.434677 | 0.126548 | 0.006696 | 0.000655 | 3 | 2000.0 | 0.0 | 8 | 1 | 200 | {'rfe__n_features_to_select': 3, 'xgb__gamma':... | -19016.356974 | -17163.287836 | -17897.134903 | -18025.593238 | 761.945958 | 72 |
75 rows × 17 columns
Now, let's generate predictions on both our training and test sets using the model trained with our best performing parameters:
y_pred_train_xgb = xgb_bs_cv.predict(X_train)
y_pred_test_xgb = xgb_bs_cv.predict(X_test)
xgb_perf_train = calc_model_performance(y_train, y_pred_train_xgb)
xgb_perf_train
{'Root Mean Squared Error': 4144.72597331536,
'Mean Squared Error': 17178753.39387496,
'Mean Absolute Error': 2345.4207978660365,
'Mean Absolute Percentage Error': 0.29464729849599547,
'R Squared': 0.8827434413879351}
xgb_perf_test = calc_model_performance(y_test, y_pred_test_xgb)
xgb_perf_test
{'Root Mean Squared Error': 4702.351935478104,
'Mean Squared Error': 22112113.725094665,
'Mean Absolute Error': 2666.388096436068,
'Mean Absolute Percentage Error': 0.3412412384406359,
'R Squared': 0.8491100641812912}
perf_comp_train = compare_model_performance(base_perf_train, xgb_perf_train)
perf_comp_test = compare_model_performance(base_perf_test, xgb_perf_test)
perf_comp_train
| base | new | abs_improvement | perc_improvement | |
|---|---|---|---|---|
| Root Mean Squared Error | 5964.03 | 4144.73 | -1819.30 | -30.50 |
| Mean Squared Error | 35569654.79 | 17178753.39 | -18390901.40 | -51.70 |
| Mean Absolute Error | 4583.19 | 2345.42 | -2237.77 | -48.83 |
| Mean Absolute Percentage Error | 0.75 | 0.29 | -0.46 | -61.33 |
| R Squared | 0.76 | 0.88 | 0.12 | 15.79 |
perf_comp_test
| base | new | abs_improvement | perc_improvement | |
|---|---|---|---|---|
| Root Mean Squared Error | 5752.48 | 4702.35 | -1050.13 | -18.26 |
| Mean Squared Error | 33090996.22 | 22112113.73 | -10978882.49 | -33.18 |
| Mean Absolute Error | 4534.42 | 2666.39 | -1868.03 | -41.20 |
| Mean Absolute Percentage Error | 0.76 | 0.34 | -0.42 | -55.26 |
| R Squared | 0.77 | 0.85 | 0.08 | 10.39 |
We can observe a significant decrease the errors and an increase in the $R^2$ value for the XGBoost model.
Most importantly, the RMSE (our model evaluation metric that we defined at the beginning of the exercise) has decreased by ~22% on the test set!